{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python3入门到精通实战特训（第19期）第4课书面作业\n",
    "学号：116646  \n",
    "**作业内容：**  \n",
    "使用多线程读写同一个文件。多个循环写线程，多个用于循环读线程；所有线程同时启动，避免重复读写。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 作业思路\n",
    "1. 创建一个指定长度初始数据源。以4096大小的块为单位将数据源分成多个块的组合。  \n",
    "2. 创建3个写线程。写线程随机从初始数据源读取数据块，并写入目标文件对应的位置。  \n",
    "3. 创建3个读线程。读线程从已经写入目标文件中的块中读取数据。\n",
    "4. 将完成读取的数据组合成一个整体，比较这个整体与初始数据源，如果完全一致，表示成功。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-16T08:48:51.438170Z",
     "iopub.status.busy": "2022-12-16T08:48:51.437732Z",
     "iopub.status.idle": "2022-12-16T08:48:51.444567Z",
     "shell.execute_reply": "2022-12-16T08:48:51.443214Z",
     "shell.execute_reply.started": "2022-12-16T08:48:51.438140Z"
    }
   },
   "outputs": [],
   "source": [
    "import time\n",
    "from threading import Thread, Lock\n",
    "import random\n",
    "\n",
    "fn = 'source.txt' # 初始数据源\n",
    "raw_len = 2876543  # 初始数据源长度"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. 创建初始数据源\n",
    "随机产生以小写字母为基础的数据源。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-16T08:48:53.608285Z",
     "iopub.status.busy": "2022-12-16T08:48:53.607893Z",
     "iopub.status.idle": "2022-12-16T08:48:59.433786Z",
     "shell.execute_reply": "2022-12-16T08:48:59.432179Z",
     "shell.execute_reply.started": "2022-12-16T08:48:53.608254Z"
    }
   },
   "outputs": [],
   "source": [
    "with open(fn, 'w') as f:\n",
    "    for i in range(raw_len):\n",
    "        c = chr(ord('a')+random.randint(0, 25))\n",
    "        f.write(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. 全局数据装备"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-16T08:49:10.341328Z",
     "iopub.status.busy": "2022-12-16T08:49:10.340357Z",
     "iopub.status.idle": "2022-12-16T08:49:10.364730Z",
     "shell.execute_reply": "2022-12-16T08:49:10.363392Z",
     "shell.execute_reply.started": "2022-12-16T08:49:10.341293Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "初始数据源大小： 2876543\n",
      "总共有703块，最后一块的大小为1151\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "2876543"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "buffer = b'' #初始数据源缓存区\n",
    "with open(fn, 'rb') as f:\n",
    "    buffer = f.read()\n",
    "buffer_len = len(buffer) # 初始数据源大小\n",
    "print('初始数据源大小：', buffer_len)\n",
    "block_size = 4096\n",
    "last_block_size = buffer_len % block_size\n",
    "block_num = buffer_len // block_size if last_block_size == 0 else buffer_len // block_size + 1\n",
    "print('总共有{}块，最后一块的大小为{}'.format(block_num, last_block_size))\n",
    "dbuffer = [] # 恢复数据源\n",
    "block_list = [i for i in range(block_num)] #待处理块列表\n",
    "random.shuffle(block_list) #随机化\n",
    "dset = {}  #读取暂存缓冲字典\n",
    "ready_to_reader = [] #可以读取块列表\n",
    "block_read = 0 #n已经读取的块数\n",
    "rdlock = Lock() #读队列锁\n",
    "flock = Lock() #文件锁\n",
    "wrlock = Lock() #写队列锁\n",
    "bflock = Lock() #源数据锁\n",
    "\n",
    "target = open('target.txt', 'w+b') #创建目标数据文件\n",
    "target.write(b'0' * buffer_len)    #指定大小为目标大小"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. 封装数据接口\n",
    "该接口已经加锁，对线程安全"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-16T08:49:16.412122Z",
     "iopub.status.busy": "2022-12-16T08:49:16.411702Z",
     "iopub.status.idle": "2022-12-16T08:49:16.419829Z",
     "shell.execute_reply": "2022-12-16T08:49:16.418544Z",
     "shell.execute_reply.started": "2022-12-16T08:49:16.412091Z"
    }
   },
   "outputs": [],
   "source": [
    "def get_data(b):\n",
    "    bflock.acquire()\n",
    "    d = buffer[b*block_size:(b+1)*block_size]\n",
    "    bflock.release()\n",
    "    return d\n",
    "\n",
    "def data_pop(lst, lock):\n",
    "    lock.acquire()\n",
    "    if len(lst)>0:\n",
    "        s = lst.pop(0)\n",
    "    else:\n",
    "        s = -1\n",
    "    lock.release()\n",
    "    return s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. 读线程执行函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-16T08:49:18.946661Z",
     "iopub.status.busy": "2022-12-16T08:49:18.946269Z",
     "iopub.status.idle": "2022-12-16T08:49:18.953896Z",
     "shell.execute_reply": "2022-12-16T08:49:18.952746Z",
     "shell.execute_reply.started": "2022-12-16T08:49:18.946630Z"
    }
   },
   "outputs": [],
   "source": [
    "def reader():\n",
    "    global target, block_read, block_num\n",
    "    while block_read < block_num:\n",
    "        b = data_pop(ready_to_reader,rdlock)\n",
    "        if b >= 0:\n",
    "            block_read += 1\n",
    "            flock.acquire()\n",
    "            target.seek(b*block_size)\n",
    "            d = b''\n",
    "            if b!=block_num:\n",
    "                d = target.read(block_size)\n",
    "            else:\n",
    "                d = target.read(last_block_size)\n",
    "            flock.release()\n",
    "            dset[b] = d\n",
    "        else:\n",
    "            time.sleep(0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. 写线程执行函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-16T08:49:21.505009Z",
     "iopub.status.busy": "2022-12-16T08:49:21.504610Z",
     "iopub.status.idle": "2022-12-16T08:49:21.511875Z",
     "shell.execute_reply": "2022-12-16T08:49:21.510559Z",
     "shell.execute_reply.started": "2022-12-16T08:49:21.504975Z"
    }
   },
   "outputs": [],
   "source": [
    "def writer():\n",
    "    global target\n",
    "    b = data_pop(block_list, wrlock)\n",
    "    while b>=0:\n",
    "        d = get_data(b)\n",
    "        \n",
    "        flock.acquire()\n",
    "        target.seek(b*block_size)\n",
    "        target.write(d)\n",
    "        target.flush()\n",
    "        flock.release()\n",
    "        \n",
    "        rdlock.acquire()\n",
    "        ready_to_reader.append(b)\n",
    "        rdlock.release()\n",
    "        \n",
    "        b = data_pop(block_list, wrlock)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. 创建线程并执行"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-16T08:49:24.058672Z",
     "iopub.status.busy": "2022-12-16T08:49:24.058281Z",
     "iopub.status.idle": "2022-12-16T08:49:24.283615Z",
     "shell.execute_reply": "2022-12-16T08:49:24.282435Z",
     "shell.execute_reply.started": "2022-12-16T08:49:24.058642Z"
    }
   },
   "outputs": [],
   "source": [
    "n_readers = 3\n",
    "n_writers = 3\n",
    "threads = [Thread(target=reader) for i in range(n_readers)] \n",
    "threads += [Thread(target=writer) for i in range(n_writers)]\n",
    "for t in threads:\n",
    "    t.start()\n",
    "for t in threads:\n",
    "    t.join()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. 比较执行结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-16T08:49:27.974253Z",
     "iopub.status.busy": "2022-12-16T08:49:27.973787Z",
     "iopub.status.idle": "2022-12-16T08:49:27.985476Z",
     "shell.execute_reply": "2022-12-16T08:49:27.984069Z",
     "shell.execute_reply.started": "2022-12-16T08:49:27.974219Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "读取数据与初始源数据完全一致！\n"
     ]
    }
   ],
   "source": [
    "dbuffer = b''.join([dset[y] for y in sorted([x for x in dset])])\n",
    "if dbuffer == buffer:\n",
    "    print(\"读取数据与初始源数据完全一致！\")\n",
    "else:\n",
    "    print(\"读取数据与初始源数据不一致！\")\n",
    "target.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.11.0 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  },
  "vscode": {
   "interpreter": {
    "hash": "854f9ff9f945bab8ba5d04e939504df285fc62eb45b7a687b185fca2375459b7"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
